GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-training

Most of the currently existing vision and language pre-training (VLP) methods have mainly focused on how to extract align text features. In contrast mainstream VLP methods, we highlight that two routinely applied steps during crucial impact performance pre-trained model: in-batch hard negative sampling for image-text matching (ITM) assigning large masking probability masked modeling (MLM). After empirically showing unexpected effectiveness above steps, systematically devise our GRIT-VLP, which adaptively samples mini-batches more effective mining ITM while maintaining computational cost pre-training. Our method consists three components: 1) GRouped mIni-baTch (GRIT) strategy collects similar examples in a mini-batch, 2) ITC consistency loss improving ability, 3) enlarged MLM. Consequently, show GRIT-VLP achieves new state-of-the-art various downstream tasks with much less cost. Furthermore, demonstrate model is essentially par ALBEF, previous state-of-the-art, only one-third training epochs same data. Code available at https://github.com/jaeseokbyun/GRIT-VLP .